IE510 Term Paper: Stochastic Gradient Descent, Weighted Sampling, and the Randomized Kaczmarz Algorithm

In this paper, we mainly study the convergence properties of stochastic gradient descent (SGD) as described in Needell et al. [2]. The function to be minimized with SGD is assumed to be strongly convex. Also, its gradients are assumed to be Lipschitz continuous. First, we discuss the superior bound on convergence (of standard SGD) obtained by Needell et al. [2] as opposed to the previous work of Bach and Moulines [1]. Then, we show that this bound can be further improved if SGD is performed with importance (weighted) sampling instead of uniform sampling. Finally, we study two applications: Logistic Regression and Kaczmarz Algorithm and demonstrate faster convergence obtained from SGD with weighted (or partially weighted) sampling. 1 Stochastic Gradient Descent In stochastic gradient descent (SGD), we minimize a function F (w) using stochastic gradients in the update rule at each step. The stochastic gradients g are such that their expectation is the gradient of F (w), E[g] = ∇F (w). Now, if F (w) = Ei∼D[fi(w)], then we have g = ∇fi(w). The update rule then is as follows, wk+1 = wk − γ∇fik(w), (1) where γ is the step size and ik is drawn i.i.d from some distribution D. In this paper, we will study the convergence of SGD on the function F , with the following assumptions 1. Each fi is continuously differentiable and the gradient of fi has a Lipschitz constant Li, i.e. ‖∇fi(w1)−∇fi(w2)‖2 ≤ Li‖w1 − w2‖2 2. F is strongly convex with parameter μ, i.e. F (w1) ≥ F (w2) +∇F (w2) (w1 − w2) + μ2 ‖w1 − w2‖ 2 2, or equivalently (w1 − w2) (∇F (w1)−∇F (w2)) ≥ μ‖w1 − w2‖2